Multi-robot cooperative platform : a task-oriented teleoperation paradigm

This thesis proposes the study and development of a teleoperation system based on multi-robot cooperation under the task oriented teleoperation paradigm: Multi-Robot Cooperative Paradigm, MRCP. In standard teleoperation, the operator uses the master devices to control the remote slave robot arms. These arms reproduce the desired movements and perform the task. With the developed work, the operator can virtually manipulate an object. MRCP automatically generates the arms orders to perform the task. The operator does not have to solve situations arising from possible restrictions that the slave arms may have. The research carried out is therefore aimed at improving the accuracy teleoperation tasks in complex environments, particularly in the field of robot assisted minimally invasive surgery. This field requires patient safety and the workspace entails many restrictions to teleoperation. MRCP can be defined as a platform composed of several robots that cooperate automatically to perform a teleoperated task, creating a robotic system with increased capacity (workspace volume, accessibility, dexterity ...). The cooperation is based on transferring the task between robots when necessary to enable a smooth task execution. The MRCP control evaluates the suitability of each robot to continue with the ongoing task and the optimal time to execute a task transfer between the current selected robot and the best candidate to continue with the task. From the operator¿s point of view, MRCP provides an interface that enables the teleoperation though the task-oriented paradigm: operator orders are translated into task actions instead of robot orders. This thesis is structured as follows: The first part is dedicated to review the current solutions in the teleoperation of complex tasks and compare them with those proposed in this research. The second part of the thesis presents and reviews in depth the different evaluation criteria to determine the suitability of each robot to continue with the execution of a task, considering the configuration of the robots and emphasizing the criterion of dexterity and manipulability. The study reviews the different required control algorithms to enable the task oriented telemanipulation. This proposed teleoperation paradigm is transparent to the operator. Then, the Thesis presents and analyses several experimental results using MRCP in the field of minimally invasive surgery. These experiments study the effectiveness of MRCP in various tasks requiring the cooperation of two hands. A type task is used: a suture using minimally invasive surgery technique. The analysis is done in terms of execution time, economy of movement, quality and patient safety (potential damage produced by undesired interaction between the tools and the vital tissues of the patient). The final part of the thesis proposes the implementation of different virtual aids and restrictions (guided teleoperation based on haptic visual and audio feedback, protection of restricted workspace regions, etc.) using the task oriented teleoperation paradigm. A framework is defined for implementing and applying a basic set of virtual aids and constraints within the framework of a virtual simulator for laparoscopic abdominal surgery. The set of experiments have allowed to validate the developed work. The study revealed the influence of virtual aids in the learning process of laparoscopic techniques. It has also demonstrated the improvement of learning curves, which paves the way for its implementation as a methodology for training new surgeons.Aquesta tesi doctoral proposa l'estudi i desenvolupament d'un sistema de teleoperació basat en la cooperació multi-robot sota el paradigma de la teleoperació orientada a tasca: Multi-Robot Cooperative Paradigm, MRCP. En la teleoperació clàssica, l'operador utilitza els telecomandaments perquè els braços robots reprodueixin els seus moviments i es realitzi la tasca desitjada. Amb el treball realitzat, l'operador pot manipular virtualment un objecte i és mitjançant el MRCP que s'adjudica a cada braç les ordres necessàries per realitzar la tasca, sense que l'operador hagi de resoldre les situacions derivades de possibles restriccions que puguin tenir els braços executors. La recerca desenvolupada està doncs orientada a millorar la teleoperació en tasques de precisió en entorns complexos i, en particular, en el camp de la cirurgia mínimament invasiva assistida per robots. Aquest camp imposa condicions de seguretat del pacient i l'espai de treball comporta moltes restriccions a la teleoperació. MRCP es pot definir com a una plataforma formada per diversos robots que cooperen de forma automàtica per dur a terme una tasca teleoperada, generant un sistema robòtic amb capacitats augmentades (volums de treball, accessibilitat, destresa,...). La cooperació es basa en transferir la tasca entre robots a partir de determinar quin és aquell que és més adequat per continuar amb la seva execució i el moment òptim per realitzar la transferència de la tasca entre el robot actiu i el millor candidat a continuar-la. Des del punt de vista de l'operari, MRCP ofereix una interfície de teleoperació que permet la realització de la teleoperació mitjançant el paradigma d'ordres orientades a la tasca: les ordres es tradueixen en accions sobre la tasca en comptes d'estar dirigides als robots. Aquesta tesi està estructurada de la següent manera: Primerament es fa una revisió de l'estat actual de les diverses solucions desenvolupades actualment en el camp de la teleoperació de tasques complexes, comparant-les amb les proposades en aquest treball de recerca. En el segon bloc de la tesi es presenten i s'analitzen a fons els diversos criteris per determinar la capacitat de cada robot per continuar l'execució d'una tasca, segons la configuració del conjunt de robots i fent especial èmfasi en el criteri de destresa i manipulabilitat. Seguint aquest estudi, es presenten els diferents processos de control emprats per tal d'assolir la telemanipulació orientada a tasca de forma transparent a l'operari. Seguidament es presenten diversos resultats experimentals aplicant MRCP al camp de la cirurgia mínimament invasiva. En aquests experiments s'estudia l'eficàcia de MRCP en diverses tasques que requereixen de la cooperació de dues mans. S'ha escollit una tasca tipus: sutura amb tècnica de cirurgia mínimament invasiva. L'anàlisi es fa en termes de temps d'execució, economia de moviment, qualitat i seguretat del pacient (potencials danys causats per la interacció no desitjada entre les eines i els teixits vitals del pacient). Finalment s'ha estudiat l'ús de diferents ajudes i restriccions virtuals (guiat de la teleoperació via retorn hàptic, visual o auditiu, protecció de regions de l'espai de treball, etc) dins el paradigma de teleoperació orientada a tasca. S'ha definint un marc d'aplicació base i implementant un conjunt de restriccions virtuals dins el marc d'un simulador de cirurgia laparoscòpia abdominal. El conjunt d'experiments realitzats han permès validar el treball realitzat. Aquest estudi ha permès determinar la influencia de les ajudes virtuals en el procés d'aprenentatge de les tècniques laparoscòpiques. S'ha evidenciat una millora en les corbes d'aprenentatge i obre el camí a la seva implantació com a metodologia d'entrenament de nous cirurgians.Postprint (published version

Assistance strategies for robotized laparoscopy

Robotizing laparoscopic surgery not only allows achieving better accuracy to operate when a scale factor is applied between master and slave or thanks to the use of tools with 3 DoF, which cannot be used in conventional manual surgery, but also due to additional informatic support. Relying on computer assistance different strategies that facilitate the task of the surgeon can be incorporated, either in the form of autonomous navigation or cooperative guidance, providing sensory or visual feedback, or introducing certain limitations of movements. This paper describes different ways of assistance aimed at improving the work capacity of the surgeon and achieving more safety for the patient, and the results obtained with the prototype developed at UPC.Peer ReviewedPostprint (author's final draft

A physical/virtual anatomical platform for hysteroscopy training

Minimally Invasive Surgery (MIS), which consists in operating through small orifices, reduces the patients’ pain, complications, recovery time and scarring. However, while providing many benefits, the skills required in MIS are especially difficult to learn. Unfortunately, current training methods for such techniques have not progressed at the same pace as surgical advances. Insufficient surgical skills acquisition translates into complications during procedures, which not only harm the patient, but also increases the derived costs of interventions and their follow up. The developed platform merges both technologies, with the benefits of having a physical anatomic model and computer assistance for augmented reality and objective assessment. Further to the functions of a surgical trainer, EndoTrainer provides an integral education, training and evaluation platform.Peer ReviewedPostprint (published version

Human machine interfaces for robot-assisted colonoscopy: a clinical survey

In the last decade, a variety of Human Machine Interfaces (HMI) have been designed for robot assisted colonoscopy, including different input devices, assistive tools and feedback. However, few studies aiming at assessing the best features of the HMI have been performed so far, leading to a lack of knowledge about the optimal HMI able to minimize the cognitive and physical load of the operators and maximize their performace. Herein, we present the results of a survey administered to more than 70 endoscopists across different European countries, to get insights about the clinicians’ desires on the next generation HMI for robot assisted colonoscopy.This work was supported by the ATLAS project, which has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement No 813782.Peer ReviewedPreprin

EndoTrainer: a novel hybrid training platform for endoscopic surgery

Purpose Endoscopy implies high demanding procedures, and their practice requires structured formation curricula supported by adequate training platforms. Physical platforms are the most standardised solution for surgical training, but over the last few years, virtual platforms have been progressively introduced. This research work presents a new hybrid, physic-virtual, endoscopic training platform that exploits the benefits of the two kind of platforms combining realistic tools and phantoms together with the capacity of measuring all relevant parameters along the execution of the exercises and of providing an objective assessment performance. Methods The developed platform, EndoTrainer, has been designed to train and assess surgical skills in hysteroscopy and cystoscopy following a structured curricula. The initial development and validation is focused on hysteroscopic exercises proposed in the Gynaecological Endoscopic Surgical Education and Assessment (GESEA) Certification Programme from The Academy and European Society for Gynaecological Endoscopy (ESGE) and analyses the obtained results of an extensive study with 80 gynaecologists executing 30 trials of the standard 30 degree endoscope navigation exercise. Results The experiments demonstrate the benefits of the presented hybrid platform. Multi-variable statistical analysis points out that all subjects have obtained statistically significant improvement in all relevant parameters: shorter and safer trajectories, improved 30-degree endoscope navigation, accurate positioning over the targets and reduction of the execution time. Conclusion This paper presents a new hybrid approach for training, and evaluating whether it provides an objectivable improvement of camera navigation endoscopic basic skills. The obtained results demonstrate the initial hypothesis: all subjects have improved their camera handling and navigation skills.Peer ReviewedPostprint (published version

SENTISIM: A hybrid training platform for SLNB in local melanoma staging

Sentinel Lymph Node Biopsy (SLNB) has been established as an accurate procedure for regional melanoma staging. SNLB has a diagnostic significance as biopsy-based staging identifies patients with nodal metastases who may benefit from immediate complete lymphadenectomy and prolongs disease-free survival for all patients. This work presents a new training platform for SLNB in local melanoma staging, that solves the previous problemsmaintaining a realistic scenario, at the same time that measures a series of important parameters to determine the quality of the surgery.Peer ReviewedPostprint (published version

Vision based robot assistance in TTTS fetal surgery

© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.This paper presents an accurate and robust tracking vision algorithm for Fetoscopic Laser Photo-coagulation (FLP) surgery for Twin-Twin Transfusion Syndrome (TTTS). The aim of the proposed method is to assist surgeons during anastomosis localization, coagulation and review using a tele-operated robotic system. The algorithm computes the relative position of the fetoscope tool tip with respect to the placenta, via local vascular structure registration.The algorithm uses image features (local superficial vascular structures of the placenta’s surface) to automatically match consecutive fetoscopic images. It is composed of three sequential steps: image processing (filtering, binarization and vascular structures segmentation); relevant Points Of Interest (POIs) seletion; and image registration between consecutive images.The algorithm has to deal with the low quality of fetoscopic images, the liquid and dirty environment inside the placenta jointly with the thin diameter of the fetoscope optics and low amount of environment light reduces the image quality. The obtained images are blurred, noisy and with very poor color components.The tracking system has been tested using real video sequences of FLP surgery for TTTS. The computational performance enables real time tracking, locally guiding the robot over the placenta’s surface with enough accuracy.Peer ReviewedPostprint (author's final draft

Intraamniotic sealing of fetoscopic membrane defects in ex vivo and in vivo sheep models using an integrated semirigid bioadhesive patch

BACKGROUND: Preterm prelabor rupture of membranes is the most frequent complication of fetoscopic surgery. Strategies to seal the membrane defect created by fetoscopy have been attempted with little success. We previously developed an integrated semirigid bioadhesive patch composed of silicone and hydroxypropyl methylcellulose that achieved ex vivo sealing of membrane defects. OBJECTIVE: To evaluate the feasibility of the insertion of our integrated semirigid bioadhesive patches using a fetoscopic technique and to test the adhesion in ex vivo human membranes and in an in vivo ovine model. STUDY DESIGN: An experimental study involving 2 experiments: (1) ex vivo—human fetal membranes were mounted in a custom-designed model with saline solution simulating intraamniotic pressure. The insertion of 2 different bioadhesive patches made of silicone-hydroxypropyl methylcellulose and silicone-polyurethane-hydroxypropyl methylcellulose was performed through a 12-Fr cannula mimicking fetoscopic surgery technique. The experiment was repeated 10 times with membranes from different donors. Measures included insertion time, successful insertion, and adhesion at 5 minutes; (2) in vivo—16 patches of silicone-hydroxypropyl methylcellulose were inserted by fetoscopy in the amniotic cavity of pregnant sheep (4 bioadhesives per animal, in 4 ewes). Measures included successful insertion, adhesion at 5 minutes, and adhesion at the end of surgery. RESULTS: In the ex vivo insertion study, there was no difference in the insertion time between silicone-hydroxypropyl methylcellulose and silicone-polyurethane-hydroxypropyl methylcellulose patches (P=.49). Insertion was successful in all cases, but complete adhesion at 5 minutes was superior for silicone-hydroxypropyl methylcellulose (P=.02). In the in vivo study, insertion of silicone-hydroxypropyl methylcellulose by fetoscopy was feasible and successful in all cases, and no complications were reported. Adhesion persisted at 5 minutes and at the end of the surgery in 68.8% and 56.3% of the patches, respectively. CONCLUSION: We describe the feasibility of deploying through a fetoscopic trocar a semirigid silicone-hydroxypropyl methylcellulose patch that seals fetal membranes after an invasive fetal procedure. The results warrant further research for improving long-term adhesion and developing a clinically applicable system.This project has been funded by the Cellex Foundation and the Erasmus+ Programme of the European Union (Framework Agreement number: 2013-0040). This publication reflects only the authors’ views, and the Commission cannot be held responsible for any use that may be made of the information contained therein. T.M. was supported by a predoctoral grant from Erasmus Mundus FetalMed-PhD. E.E. has received funding from the Departament de Salut under grant number SLT008/18/00156.Peer ReviewedPostprint (published version

Multi-level-assistance robotic platform for navigation in the urinary system: design and preliminary tests

This work was supported by the ATLAS project. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 813782. This work was also partially supported by French State Funds managed by the “Agence Nationale de la Recherche (ANR)” through the “Investissements d’Avenir” (Investments for the Future) Program under Grant ANR-10-IAHU-02 (IHU-Strasbourg).Peer ReviewedPostprint (published version

Optimization criterion for safety task transfer in cooperative robotics

This paper presents a strategy for a cooperative multirobot system, constituting a Virtual Robot. The Virtual Robot is composed of a set of robotic arms acting as only one, transferring the execution of a teleoperated task from one to another when necessary. To decide which of the robots is the most suitable to execute the task at every instant, a multiparametric decision function has been defined. This function is based on a set of intrinsic and extrinsic evaluation indexes of the robot. Since the internal operation of the Virtual Robot must be transparent to the user, a control architecture has been developed.Postprint (published version